Image processing method, image processing apparatus, and image processing program

ABSTRACT

An image feature analysis section performs image feature analysis of an input image to generate an image feature quantity independent from image coordinates. An image content densification section information densifies the image feature quantity to generate a densified image feature quantity. An image generation section generates, based on the densified image feature quantity, an image in which the information content of the input image is densified.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of Application PCT/JP2004/019374, filed on Dec.24, 2004.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing for imagemeasurement/recognition and image generation, and particularly relatesto a technology for generating information densified images of whichinformation contents are greater than those of the original images.

2. Description of the Related Art

In recent years, digitalization of various kinds of image processingequipment, video equipment, and the like, and widespread use of Internetlead to interconnection of computers and household electricalappliances, so that generally-called ubiquitous network societyemploying remote control, media merger, and the like has been beingorganized. The Specs of image equipment are wide-ranged due todifference in mechanism, portability, function, and the like, andtherefore, image information in various formats is distributed. In aone-segment terrestrial digital broadcast for mobile phones, forexample, down-conversion of an HDTV (High Definition Television) imageof 1920 pixels×1080 lines to an image of, for example, 320 pixels×240lines is required for display on the display of a mobile phone. Inshort, the spatial resolution must be converted according to the spec ofimage equipment. Further, there may be the case where time resolution inwhich difference is caused due to difference in refresh rate may beconverted in addition to the spatial resolution. For example, a telecineprocess for converting a movie film format of 24 frames per second intoa video format of 30 frames per second is raised as one example.

In resolution conversion, generation of data of which resolution isgreater than that at sampling is called “super-resolution.” For example,displaying an image recorded in DV format (576 pixels×480 lines) as anHDTV image requires super-resolution processing for increasing thepixels about two times and the lines about 2.5 times. Also, ahigh-resolution script is needed for printing. For example, printing onA4 paper (297 mm×210 mm) at a resolution of 600 dpi requires preparationof a script of 7128 pixels×5040 lines. Almost of all cameras have aresolution lower than it, and accordingly, the super-resolutionprocessing is essential in, for example, direct printing from a digitalstill camera to a printer. The aforementioned two examples use pixelcount and line count for representing the resolution and is defined as“space-domain super-resolution.” On the other hand, there may be a caserequiring “time-domain super-resolution” for increasing time resolutionto be higher than that at sampling. For example, displaying an imagerecorded by a branch scanning (interlace) method on a display bysequential scanning (progressive) method requires time-domainsuper-resolution processing for increasing the time resolution twotimes. Such processing is used in various cases such as a case for usinganalog broadcast material in digital broadcast.

Such super-resolution processing is regarded as interpolation forgenerating new data from existing data. The principal conception of theinterpolation is estimation of new data from existing data present inthe vicinity of the new data. In the space-domain super-resolution,signal values of new data are estimated from signal values of pixelshorizontally, perpendicularly, or obliquely adjacent thereto. In thetime-domain super-resolution, new data is estimated from immediatelypreceding data and next data. As specific methods for the space-domainsuper-resolution, a nearest neighbor interpolation, a bi-linearinterpolation, bi-cubic interpolation, and the like are generally known(Non-patent Document 1: “Clarify Three-dimensional Computer Graphics,”Sinji Araya, published by Kyoritsu Shuppan, Sep. 25, 2003, pp. 144-145).Further, there has been proposed compensation for quality degradation,which is due to blur in these interpolation methods, by supplementinghigh frequency component (Patent Document 1: Japanese Patent ApplicationLaid Open Publication No. 2003-018398A (FIG. 2)).

On the other hand, there has been proposed a method for realizingsuper-resolution processing in a manner that much low-resolution dataare collected so as to include an overlapping region and thecorresponding points are connected (Patent Document 2: Japanese PatentApplication Laid Open Publication No. 10-069537A (FIG. 2)).

SUMMARY OF THE INVENTION

The super-resolution processing techniques disclosed in Non-patentDocument 1 and Patent Document 1 as mentioned above, however, ignorepatterns, gloss and the like in the surfaces of objects, and nomechanism for maintaining patterns, gloss, and the like of the originalimages in images that have been subjected to the super-resolutionprocessing is included. In this connection, texture impression of theoriginal images shall be degraded by the super-resolution processing andthe texture feelings about the objects in the images may becomedifferent.

Also, the technique in Patent Document 2 requires image shooting pluraltimes, resulting in increase in number of working processes.

The present invention has been made in view of the above problems andhas its objective of providing an image processing technique forgenerating an image of which information content is greater than that ofthe original image with no degradation in image feature of the originalimage.

In order to solve the above problems, in the present invention, an imagefeature (e.g., density distribution, frequency distribution, contrast,and the like) of the original image is analyzed, and informationdensification processing for increasing the image content is performedusing the thus analyzed image feature and an image feature obtained froman information densified image of which information content (e.g., pixelcount, gradation count, color channel count, and the like) is greaterthan that of the original image. Especially, when resolution is employedas the image information content, the texture (a general term of anattribute such as patterns, gloss and the like) of an input image isanalyzed, and a space-domain super-resolution image or a time-domainsuper-resolution image is generated using the thus analyzed texturefeature quantity and super-resolution texture feature quantity obtainedfrom a super-resolution image having a higher spatial or timeresolution.

According to the present invention, an information densified image ofwhich image feature is maintained with no degradation of image featureof the original image and of which image information content is greaterthan that of the original image can be generated. Especially, whenresolution is employed as the image information content, a space-domainsuper-resolution image or a time-domain super-resolution image can begenerated while maintaining impression of the texture that the originalimage has.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the constitution of an imageprocessing apparatus according to the first embodiment of the presentinvention.

FIG. 2 shows one example of the constitution of the informationdensification section in FIG. 1.

FIG. 3 is a conceptual drawing of space-domain super-resolutionprocessing according to the first embodiment of the present invention.

FIG. 4 is a block diagram illustrating the schematic constitution of animage processing apparatus for performing the space-domainsuper-resolution processing according to the first embodiment of thepresent invention.

FIG. 5 is a drawing showing one example of the constitution and theprocessing of the texture analysis section in FIG. 4.

FIG. 6 is a drawing showing one example of the constitution and theprocessing of the super-resolution processing section in FIG. 4.

FIG. 7 is a drawing showing one example of the constitution and theprocessing of the image generation section in FIG. 4.

FIG. 8 is a block diagram illustrating the constitution of an imageprocessing apparatus according to the second embodiment of the presentinvention.

FIG. 9 is a block diagram illustrating the schematic constitution of animage processing apparatus for performing the space-domainsuper-resolution processing according to the second embodiment of thepresent invention.

FIG. 10 is a drawing showing one example of a method of calculating abasic texture feature quantity weight coefficient in the constitution ofFIG. 9.

FIG. 11 is a drawing illustrating the constitution of an imageprocessing apparatus according to the third embodiment of the presentinvention.

FIG. 12 is a drawing showing one example of a method of structuring atexture feature vector conversion table.

FIG. 13 is a drawing showing the first constitutional example of thepresent invention.

FIG. 14 is a drawing showing the second constitutional example of thepresent invention.

FIG. 15 is a drawing showing the third constitutional example of thepresent invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The first aspect of the present invention provides an image processingmethod including: a first step of performing image feature analysis foran original image to obtain an image feature quantity independent fromimage coordinates; a second step of subjecting the image featurequantity obtained in the first step to information densification toobtain a densified image feature quantity; and a third step ofgenerating, based on the densified image feature quantity obtained inthe second step, a densified image in which an information content ofthe original image is densified.

The second aspect of the present invention provides the image processingmethod of the first aspect, wherein the second step includes the stepsof: selecting an image feature category to which the image featurequantity belongs from a plurality of image feature categories preparedin advance; and reading out a basic image feature quantity, of whichinformation content is densified, in the selected image feature categoryas the densified image feature quantity from a densified image featuredatabase.

The third aspect of the present invention provides the image processingmethod of the first aspect, wherein the second step includes the stepsof: calculating respective similarities of the image feature quantity toplural image feature categories prepared in advance; and summingrespective basic image feature quantities in the plural image featurecategories with weights according to the respective calculatedsimilarities to generate the densified image feature quantity.

The fourth aspect of the present invention provides the image processingmethod of the first aspect, wherein the second step includes the stepsof: selecting an image feature category to which the image featurequantity belongs from a plurality of image feature categories preparedin advance; and converting the image feature quantity to the densifiedimage feature quantity by referencing a conversion table database andusing a conversion table for feature quantity conversion in the selectedimage feature category.

The fifth aspect of the present invention provides the image processingmethod of the fourth aspect, wherein the plurality of image featurecategories are provided for respective material properties of an objectshot in an image.

The sixth aspect of the present invention provides the image processingmethod of the first aspect, wherein spatial resolution or timeresolution is used as the image feature quantity.

The seventh aspect of the present invention provides the imageprocessing method of the sixth aspect, wherein spatial frequencyresponse or time frequency response is obtained using Fourier transform.

The eighth aspect of the present invention provides the image processingmethod of the sixth aspect, wherein spatial frequency response or timefrequency response is obtained using wavelet transform.

The ninth aspect of the present invention provides the image processingmethod of the sixth aspect, wherein spatial frequency response or timefrequency response is obtained using a plurality of spatial filters ofwhich at least one of scale, phase, and spatial directionality isdifferent from each other.

The tenth aspect of the present invention provides an image processingapparatus including: an image feature analysis section that performsimage feature analysis for an original image to obtain an image featurequantity independent from image coordinates; an informationdensification section that performs information densification for theimage feature quantity obtained by the image feature analysis section toobtain a densified image feature quantity; and an image generationsection that generates, based on the densified image feature quantityobtained by the information densification section, a densified image inwhich an information content of the original image is densified.

The eleventh aspect of the present invention provides an imageprocessing program that allows a computer to execute: a first step ofperforming image feature analysis for an original image to obtain animage feature quantity independent from image coordinates; a second stepof subjecting the image feature quantity obtained in the first step toinformation densification to obtain a densified image feature quantity;and a third step of generating, based on the densified image featurequantity obtained in the second step, a densified image in which aninformation content of the original image is densified.

The preferred embodiments of the present invention will be describedbelow with reference to the drawings.

First Embodiment

FIG. 1 is a block diagram illustrating the constitution of an imageprocessing apparatus according to the first embodiment of the presentinvention. The image processing apparatus shown in FIG. 1 generates aninformation densified image of which information content is greater thanthat of the original image.

Herein, “information densification” in the present description meansprocessing for increasing the image information content of a providedimage and is called, under certain circumstances, “super informationprocessing” or “super computerization.” The term, image informationcontent represents pixel count, gradation count, color channel count,and the like, for example. Referring to pixel count, for example,information densification for expanding an image of 320 pixels×240 linesfour times in both the horizontal and perpendicular directions generatesan image of 1280 pixels×960 lines of which total pixel count is 16times. In a case of gradation count, the processing for expanding aninput image in which each pixel has 128 gradations to be an image inwhich each pixel has 256 gradations corresponds to two-time informationdensification. In a case of color channel count, conversion of amonochrome image (one color channel) to an RGB image corresponds tothree-time information densification.

Conversion in combination of the above three examples, that is,conversion of “a monochrome image of 320 pixels×240 lines, each pixelbeing composed of one color channel with 128 gradations” to “an RBGimage of 1280 pixels×960 lines, each pixel being composed of three colorchannels with 256 gradations” corresponds to information densificationfor increasing the image information content 96 times (16×2×3).

Referring to the constitution in FIG. 1, an image feature analysissection 10 analyzes an image feature of an input image IIN as theoriginal image and outputs an image feature quantity FI (first step).Herein, the image feature means, for example, density distribution,frequency response, contrast, and the like, and they are expressed by adensity histogram, a frequency spectrum, a ratio of a highlighted partto a dark part, respectively. An information densification section 20performs information densification based on the image feature quantityFI and outputs a densified image feature quantity SFI (second step).Direct information densification of the image feature quantity FIrealizes image content densification of an image with no degradation ofthe image feature itself. An image generation section 30 visualizes thedensified image feature quantity SFI to generate an output image IOUT asa densified image (third step).

FIG. 2 shows one example of the constitution of the informationdensification section 20 in FIG. 1. In FIG. 2, an image feature categoryselection section 21 selects, from image feature categories classifiedin advance, a category to which the image feature quantity FI belongsand outputs an image feature index ID indicating the kind of theselected feature quantity category. The image feature index ID isprovided to a densified image feature database 22, so that aninformation densified image feature quantity which corresponds to theimage feature index ID is output as a densified image feature quantitySFI from the densified image feature database 22.

Specifically, for example, the image feature quantity is expressed interms of vector (for example, the frequency of a density histogram isused as a vector element.), and similar vectors are gathered by anoptional clustering algorithm (e.g., K-means clustering algorithm) toform categories. Then, a category is selected by vector quantization.This is an effective algorithm. The densified image feature database 22is structured prior to execution of the information densification. Adensified sample image of which information content is greater than thatof an input image is prepared in advance for obtaining the image featurequantity of, for example, density histogram. For example, for convertingan input image of 320 pixels×240 lines to a densified image of 1280pixels×960 lines, an image of 1280 pixels×960 lines is prepared as adensified sample image. The vector expression method, the clusteringalgorithm, and the method of providing an image feature index employedin the image feature category selection section 21 and the densifiedimage feature database 22 are common.

FIG. 3 shows a conception of space-domain super-resolution processingusing spatial resolution as an image information feature. Observation ofthe density distribution on a line L in a low-resolution image X resultsin a density distribution X. For convenience in explanation, the pixelcount on the line L is set to 8 herein. Also, the density distributionreflects the data of the illustrated image not so precisely and isindicated for depicting the conception schematically. The same isapplied to the following description.

In a case of super-resolution processing for increasing the pixel countfour times, the pixel count of each line is to be 32, which necessitates32 density levels. Accordingly, compensation of density data of 24pixels by any means is necessary. In this case, a compensation methodcould be contemplated in which the density of the low-resolution image Xis arranged at regular intervals of four pixels and the pixelstherebetween are compensated by linear interpolation, as shown in thedensity distribution A, for example. In this case, however, a blurryimage like the image A is obtained due to smooth gradient while theincrease/decrease pattern of density variation along the line L ismaintained. This is the case where four-time increase in imageinformation content degrades the image feature, that is, a textureimpression.

On the other hand, the density distribution B indicates high frequencycomponent created regardless of the waveform of the density distributionX serving as low frequency component. The variation in density levelbecomes greater and more sharp than that of the density distribution A,so that fine texture like the image B is generated. However, thewaveform is rather different from that of the density distribution A,with a result of degradation of the texture impression.

The density distribution C is obtained in a manner that the densitydistribution A is maintained as the low frequency component and highfrequency component of which spatial frequency is higher than that ofthe density distribution A is superposed. In this case, the lowfrequency component traces the basic pattern of the texture and atexture pattern of which high frequency component is fine is added, sothat short of density level for 24 pixels can be compensated whilemaintaining the texture impression.

It is noted that the above description is applicable to interpolation inthe horizontal direction of images as well, and the description thereofis omitted.

Next, a method for realizing practically “the technical conception forsuperposing the high frequency component on the low frequency component”(density distribution C) as depicted in FIG. 3 will be described withreference to FIG. 4, FIG. 5, FIG. 6, and FIG. 7.

FIG. 4 is a block diagram illustrating the schematic constitution of animage processing apparatus for performing space-domain super-resolutionprocessing using spatial resolution as the image information feature.The texture as the image feature of the input image IIN is analyzed on apixel-by-pixel basis in a texture analysis section 40 as the imagefeature analysis section, so as to be described as a texture featurevector FVT. A super-resolution processing section 50 as the informationdensification section executes the super-resolution processing in thetexture feature space to convert the texture feature vector FVT as theimage feature quantity to a super-resolved texture feature vector SFVTas a densified image feature quantity. An image generation section 60visualizes the super-resolved texture feature vector SFVT and outputsthe output image IOUT.

FIG. 5 shows one example of the constitution and the processing of thetexture analysis section 40 in FIG. 4. As shown in FIG. 5, the textureanalysis section 40 performs the texture analysis using spatialfrequency response. The input image IIN is divided into a plurality ofchannels in a special frequency component resolution section 41 to beprovided to respective spatial frequency bands, thereby obtainingspecial freqeuncy responses FRS. A texture feature vector generationsection 42 generates a texture feature vector FVT using the spatialfreqeuncy responses FRS as elements. The texture feature vector FVT hasa direction and a scale within a texture feature space with responsechannels in the spatial frequency bands as axes, and describes thetexture according to the attribute thereof. It is noted that if eachelement of the feature quantity is independent from each other, thefeature quantity is described highly efficiently with no overlappingregion. Therefore, Fourier transform and wavelet transform are effectivefor resolving the spatial frequency components. The spatial frequencyresponse may be obtained using a plurality of spatial filters of whichat least one of scale, phase and spatial directionality is differentfrom each other.

FIG. 6 shows one example of the constitution and the processing of thesuper-resolution processing section 50 in FIG. 4. In FIG. 6, asuper-resolved texture feature database 51 stores, on a category-bycategory basis of the plural image feature categories, texture featurevectors generated from a sample image of which resolution is grater thanthat of an input image IIN. To texture feature vectors, respectiveindexes 1 to M are provided for specifying the image feature category. Atexture category selection section 52 compares the texture featurevector FVT describing the texture of the input image IIN with eachtexture feature vector stored in the super-resolved texture featuredatabase 51.

Herein, the texture feature vector FVT of the input image IIN, which hasa resolution lower than that of the sample image, has no effectiveresponse in the high frequency component (response over a thresholdvalue provided arbitrarily), and a response appears in the range fromdirect-current component to intermediate frequency component (frequencyw in this case). In this connection, the texture category selectionsection 52 calculates each inner product of the vectors, specifically,inner products of low-resolution components corresponding to thosehaving response in the texture feature vectors FVT, and sets the resultsas similarities. The index (an index S in this case) having the largestinner product (the highest similarity) is selected as the texturefeature index IDT, so that the texture feature vector to which thistexture feature index IDT is provided is output as a super-resolvedtexture feature vector SFVT. The super-resolved texture feature vectorSFVT has response also in a frequency band over the frequency w, andthis corresponds to the super-resolution processing in the texturefeature space. Wherein, the response amounts are indicated in a dynamicrange of 0 to 100 in FIG. 6.

FIG. 7 shows one example of the constitution and the processing of animage generation section 60. As shown in FIG. 7, the processing hereinis reversed processing of the spatial frequency resolution depicted inFIG. 5. In detail, a basic function and a product in each band of thespatial frequency are calculated for each elemet of the super-resolvedtexture feature vector SFVT, and the sum of all the channels is outputas the output image IOUT. Wherein, in the case where Fourier transformor wavelet transform is employed in the texture analysis section 40, theimage generation section 60 executes inverse transform to the employedone.

As described above, in the present embodiment, the texture selection isperformed in a manner that the texture of an input image is described interms of spatial frequency spectrum and is compared with a spatialfrequency spectrum generated from a super-resolution sample image ofwhich resolution is greater than that of the input image. Hence, aneffect that the texture impression of the image subjected tosuper-resolution processing matches that of the input image can beobtained with reliability.

It is noted that though the space-domain super-resolution processingusing the spatial resolution as the image feature quantity is describedherein, time-domain super-resolution processing using time resolution asthe image future amount can be performed as well as the space-domainsuper-resolution processing described herein. In this case, the textureis generated from difference in level of video signals accompanied bytime variation. Accordingly, the texture analysis section 40 in FIG. 4is composed of time-domain to perform time frequency resolution. Theprocedures following the time-domain development are the same as thosedepicted in FIG. 4 to FIG. 7, and therefore, the description thereof isomitted.

Second Embodiment

FIG. 8 is a block diagram illustrating the constitution of an imageprocessing apparatus according to the second embodiment of the presentinvention. The image processing apparatus shown in FIG. 8 generates aninformation densified image of which information content is greater thanthat of the input image, similar to the image processing apparatus inFIG. 1. In FIG. 8, the same reference numerals are assigned to theelements common to those in FIG. 1, and the detailed description thereofis omitted.

An information densification section 20A includes a densified imagefeature database 25, a basic image feature quantity weight coefficientcalculation section 26, and an image feature quantity interpolationsection 27. The densified image feature database 25 stores, on acategory-by-category basis of the plural image feature categories, basicimage feature quantities which are densified in information content andare generated from a densified sample image of which information contentis greater than that of the input image IIN. The basic image featurequantity weight coefficient calculation section 26 calculates eachsimilarity between the image feature quantity FI obtained from the inputimage IIN and the basic image feature quantities stored in the densifiedimage feature database 25 to obtain a basic image feature quantityweight coefficient group GWC based on the calculated similarities. Thebasic image feature quantity weight coefficient group GWC is provided tothe image feature quantity interpolation section 27. In associationtherewith, the densified image feature database 25 supplies a storedbasic image feature group GSFI to the image feature quantityinterpolation section 27. The image feature quantity interpolationsection 27 calculates weighing linear sum of the basic image featuregroup GSFI, using the basic image feature quantity weigh coefficientgroup GWC and outputs the results as densified image feature quantitiesSFI.

In short, the information densification is performed by linearinterpolation in the image feature space in the present embodiment.Accordingly, the image feature of the input image IIN is maintained inthe densified output image IOUT. Further, the interpolation of theplural basic image feature quantities using the basic image featurequantity weight coefficients enables generation of further precise imagefeature quantities of which information content is densified.

FIG. 9 is a block diagram illustrating the schematic constitution of animage processing apparatus for performing space-domain super-resolutionprocessing using spatial resolution as the image information content. InFIG. 9, the same reference numerals are assigned to the elements commonto those in FIG. 4, and the detailed description thereof is omitted.

A super-resolution processing section 50A as the informationdensification section includes a super-resolved texture feature database55, a basic texture feature quantity weight coefficient calculationsection 56, and a texture feature quantity interpolation section 57. Thesuper-resolved texture feature database 55 stores, on acategory-by-category basis of the plural image feature categories,super-resolved basic texture feature vectors, as basic image featurequantities, which are generated from super-resolved sample images ofwhich resolution is greater than that of the input image IIN. The basictexture feature quantity weight coefficient calculation section 56calculates each similarity between the texture feature vector FVTobtained from the input image IIN and the basic texture feature vectorsstored in the super-resolved texture feature database 55 to obtain abasic texture feature quantity weight coefficient group GWCT based onthe calculated similarities. The basic texture feature quantity weightcoefficient group GWCT is provided to the texture feature quantityinterpolation section 57. In association therewith, the super-resolvedtexture feature database 55 supplies a stored basic texture featurevector group GSFVT to the texture feature quantity interpolation section57. The texture feature quantity interpolation section 57 calculatesweighing linear sum of the basic texture feature vector group GSFVT,using the basic texture feature quantity weight coefficient group GWCTand outputs the results as the super-resolved texture feature vectorsSFVT.

In short, the super-resolution processing is performed by linearinterpolation in the texture feature space. Accordingly, the texture ofthe input image IIN is maintained in the super-resolved output imageIOUT. Further, the interpolation of the plural basic texture featurequantities using the basic texture feature quantity weight coefficientsenables generation of further precise image feature quantities of whichinformation content is densified.

FIG. 10 shows one example of a method of calculating a basic texturefeature quantity weight coefficient. The basic texture feature quantityweight coefficient calculation section 56 calculates similarities of thetexture feature vector FVT describing the texture of the input image IINto the basic texture feature vector group GSFVT that the super-resolvedtexture feature database 55 stores. The texture feature vector FVT hasno effective response in the high frequency component (response over athreshold value provided arbitrarily), and a response appears in therange from direct-current component to intermediate frequency component(frequency w in this case). In this connection, the basic texturefeature quantity weight coefficient calculation section 56 calculateseach inner products of the vectors, specifically, inner products oflow-resolution components corresponding to those having response in thetexture feature vector FVT, and sets the results as similarities.Subsequently, normalization is performed so that the total sum of theinner products becomes 1, and then, the results are output as a basictexture feature vector weight coefficient group GWCT. The basic texturefeature vector group GSFVT has response also in a frequency band overthe frequency w, and this corresponds to the super-resolution processingin the texture feature space. Wherein, the response amounts areindicated in a dynamic range of 0 to 100 in FIG. 10.

As described above, in the present embodiment, the super-resolvedtexture feature vector is calculated in a manner that the texture of aninput image is described in terms of spatial frequency spectrum and thebasic texture feature vectors generated from a super-resolution sampleimage of which resolution is greater than that of the input image arelinear-interpolated using the weight coefficients obtained from thesimilarities. Hence, an effect that the texture impression of the imagesubjected to super-resolution processing matches that of the input imagecan be obtained with reliability.

It is noted that though the space-domain super-resolution processingusing the spatial resolution as the image feature quantity is describedherein, time-domain super-resolution processing using time resolution asthe image future amount can be performed as well as the space-domainsuper-resolution processing described herein. In this case, the textureis generated from difference in level of video signals accompanied bytime variation. Accordingly, the texture analysis section 40 in FIG. 9is composed of time-domain to perform time frequency resolution. Theprocedures following the time-domain development are the same as thosedescribed in FIG. 9, and therefore, the description thereof is omitted.

Third Embodiment

FIG. 11 shows the constitution of an image processing apparatusaccording to the third embodiment of the present invention. The imageprocessing apparatus shown in FIG. 11 generates a super-resolution imageIOUT of which resolution is greater than that of the input image IIN,and the same reference numerals are assigned to the elements common tothose in FIG. 4. The information densification section is composed of atexture feature database 71 and a texture feature vector conversiontable 72. The texture as the image feature of the input image IIN isanalyzed on a pixel-by-pixel basis in the texture analysis section 40 tobe described as a texture feature vector FVT. The internal operation ofthe texture analysis section 40 is the same as that in FIG. 5 forgenerating the texture feature vectors FVT from spatial freqeuncyresponses FRS.

The texture feature database 71 is built from images of i levels ofresolutions and j types of material properties, namely, (i×j) sampleimages in total. The (i×j) sample images are converted to texturefeature vectors by the texture analysis section 40, and the histogramsthereof are registered on a per sample image basis into the texturefeature database 71. In detail, the texture feature vectors are obtainedon a pixel-by-pixel basis of the sample images by the texture analysissection 40 and the frequency of the texture feature vectors of all thepixels is obtained. Whereby, a plurality of image feature categoriesM_(—)1 to M_j are defined on a type-by-type basis of the materialproperties shot in the image.

One of the conditions for the super-resolution herein is that at leastone of i levels of different resolutions of the sample images is greaterthan that of the input image IIN. Also, the material properties meanwoodgrain, paper, stone, sand, and the like, for example, and may bedefined according to physical properties or the visual sense of a human.Further, referring to the woodgrain, various expressions are possible,such as a woodgrain of which surface is rough or smooth, a brightwoodgrain, and the like. Thus, the expressions regarding the types ofmaterial properties are wide ranged. The present invention does notlimit the expressions and recognizes any arbitrary definitions.

It is noted that in a case where frequencies of the elements of thefeature quantity histograms are low with less pixels having the sametexture feature vector, histogram formation by gathering similar vectorsusing a clustering algorithm (e.g., K-means clustering algorithm)reduces data amount with no degradation of the texture feature.

A feature quantity histogram obtained from the texture feature vectorsFVT of all the pixels of the input image IIN is compared with thetexture feature database 71 thus prepared in advance (wherein, histogramsimilarity comparison method is optional.). In FIG. 11, the featurequantity histogram H1 of the material property M_(—)2 and the resolutionR_(—)2 is selected as the histogram having the highest similarity to thefeature quantity histogram of the input image IIN. For super-resolutionwith no impression degradation of the texture as the image feature, afeature quantity histogram having a resolution greater than that of theinput image IIN is selected in the image feature category of the samematerial property (the material property M_(—)2 in this case). In thisexample, the feature quantity histogram H2 of the resolution R_i isselected.

It is noted that the processing is performed using the histogramsherein, so as to be applicable to the case where the space information(space coordinates) does not match between at the execution time and atthe learning time, that is, the time when the texture feature database71 is structured.

Next, the texture feature vector conversion table 72 is utilized forsuper-resolution of the texture feature quantity of the input image IIN.The texture feature vector conversion table 72 is paired with thetexture feature database 71 so as to store (i×j) conversion tables of ilevels of resolutions and j types of material properties. The featurequantity histogram H1 of “the material property M_(—)2 and theresolution R_(—)2” is selected supposing that the feature quantityhistogram H1 has the highest similarity to the texture feature vectorsFVT of the input image IIN. Accordingly, the conversion table TB of“(M_(—)2-R_(—)2) to (M_(—)2-R_i)” is referenced for converting thefeature quantity histogram H1 to the feature quantity histogram H2 of“the material property M_(—)2 and the resolution R_i”. The output of thetexture feature vector conversion table 72, which serves as asuper-resolved texture feature vector SFVT, is visualized by the imagegeneration section 60, thereby obtaining the output image IOUT.

FIG. 12 shows one example of a method of structuring the texture featurevector conversion table 72. Low-resolution images are generated by a“low-pass filtering and sub-sampling” step by step, starting from theimage having the highest resolution. Herein, the image having thehighest resolution R−i is allowed to pass through a low-pass filter 81and is reduced in resolution by the sub-sampling 82, thereby obtainingan image having a resolution R_i−1. Likewise, the image having theresolution R_i−1 is allowed to pass through a low-pass filter 83 and isreduced in resolution by the sub-sampling 84, thereby obtaining an imagehaving a resolution R_i−2.

Then, the texture analysis is performed for the respective images toobtain respective texture feature vectors. In FIG. 12, the kinds of thetexture feature vectors are expressed in terms of label number. Thus, alabeled image A of which each pixel has a label number is obtained fromthe image having the resolution R_i. Likewise, a labeled image B isobtained from the image having the resolution R_i−1, and a labeled imageC is obtained from the image having the resolution R_i−2. Wherein,respective parts of the labeled images A, B, and C are illustratedschematically because it is unnecessary to show all the pixels in thedescription herein.

The texture feature vector conversion table for converting the imagehaving the resolution R_i−1 to the image having the resolution R_i isstructured according to the correspondence in label number between thelabeled image B and the labeled image A as follows, for example. Supposethat, as indicated in “Label Correspondence Example 1”, the label number“5” is present in two pixels of the labeled image B and corresponds tothe four kinds of the texture feature vectors respectively having thelabel numbers “3”, “5”, “7”, and “8” in the labeled image A. Thefrequencies are 1, 2, 4, and 1, respectively. Under the circumstances,the texture feature vector having the label number “7” of whichfrequency is the maximum is use as a super-resolved texture featurevector. The texture feature vector conversion table can be structured bysuch simple selection processing, for example.

Moreover, weighting linear sum of each texture feature vector accordingto the frequencies realizes super-resolution according to occurrencefrequency, resulting in enhancement of texture maintenance whileincreasing the calculation amount.

In the aforementioned two methods, labels are in one-to-onecorrespondence and one texture feature vector is converted to onesuper-resolved texture feature vector. In the example in FIG. 12, onepixel of the labeled image B corresponds to four pixels of the labeledimage A. Therefore, this causes assignment of the same texture featurevector to the four pixels of the labeled image A. Nevertheless, it isdesired for effective super-resolution that different super-resolvedtexture feature vectors are assigned to the four pixels, respectively.

As a specific method for tackling this problem, it is possible to applya super-resolved texture feature vectors to the respective pixelsaccording to the respective label frequencies. In detail, among theeight pixels of the labeled image A corresponding to the label “5” ofthe labeled image B, the labels “3”, “5”, “7”, and “8” are assigned toone pixel, two pixels, four pixels, and one pixels, respectively.

Wherein, the texture pattern rarely occupies the same spatial positionat preparation of the texture feature database 71 as that at executionof the super-resolution for the input image IIN, and therefore,utilization of the spatial position information of the labeled image Ais not necessarily appropriate. In this connection, it is preferable toassign super-resolved texture feature vectors to the pixels according tothe label frequencies randomly by means of random number generation, orthe like. While pixels are selected at random, each selectedsuper-resolved texture feature vector and its frequency are determinedaccording to the label correspondence.

On the other hand, the texture feature vector conversion table forconverting the image having the resolution R_i−2 to the image having theresolution R_i is built from of a label number combination of thelabeled image C and the labeled image A. For example, the combinationand the frequency of the label number “11” of the labeled image C andthe label of the labeled image A are as indicated in “LabelCorrespondence Example 2.” There are two label numbers “7” and “9” ofwhich frequency is the maximum, and accordingly, the average of the twotexture feature vectors respectively having the labels “7” and “9” isset as a super-resolved texture feature vector. Or, the method isemployable which is already mentioned as a method of preparing theconversion table for converting the image having the resolution R_i−1 tothe image having the resolution R_i.

As described above, in the present embodiment, the texture featurequantities are super-resolved on a per material property basis, therebyrealizing the super-resolution processing with no degradation of thematerial property feelings of the texture. Further, the texture featurequantity is prepared for each of the plural resolutions, therebyrealizing the super-resolution processing with no texture degradation inboth texture selection of the low-resolution phase (i.e., an inputimage) and image generation of the high-resolution phase (i.e., anoutput image).

It is noted that all or a part of the means in the image processingapparatus according to the present invention or of steps of the imageprocessing method according to the present invention may be realizedwith the use of exclusive hardware or may be realized on software with acomputer program.

(First Constitution Example)

FIG. 13 shows the first constitution example as one example of theconstitution for performing the image processing according to thepresent invention, using a personal computer. The resolution of a camera101 is lower than the resolution of a display 102, and asuper-resolution image is generated by an image processing programaccording to the present invention loaded in a main memory 103 in orderto make the most out of the display ability of the display 102. Alow-resolution image captured by the camera 101 is recorded in an imagememory 104. A super-resolved texture feature database 105 a isstructured in advance in an external storage 105 so as to be referencedfrom the image processing program in the main memory 103. The imageprocessing program in the main memory 103 reads the low-resolution imagein the image memory 104 through a memory bus 106, converts it to ahigh-resolution image according to the resolution of the display 102,and then, transfers it to a video memory 107 through the memory bus 106gain. The high-resolution image transferred to video memory 107 isdisplayed on the display 102 to be seen. Wherein, the operation of theimage processing program, the content and the structuring method of thedatabase, and the like are any of the details mentioned in the aboveembodiments and the description thereof is omitted.

It is noted that the present invention is not limited to theconstitution of FIG. 13, and various constitutions are employable. Forexample, the super-resolved texture feature database 105 a may bereferenced through a network 108 from an external storage connected toanother personal computer. Further, the low-resolution image may beobtained through the network 108.

(Second Constitution Example)

FIG. 14 shows the second constitution example as one example of theconstitution for performing the image processing according to thepresent invention, using a server-client system. The resolution of acamera 111 is lower than the resolution of a display 112, and thesuper-resolution processing is executed in the server-client system inorder to make the most out of the display ability of the display 112. Aserver 113 includes a texture analysis section 114 and asuper-resolution processing section 115, and super-resolves the texturefeature quantity FT of the input image IIN to send the result as asuper-resolved texture feature quantity SFT to a client 117 through anetwork 116. The client 117 visualizes the received super-resolvedtexture feature quantity SFT in an image generation circuit 118, anddisplays the thus obtained super-resolution image on the display 112.Wherein, the texture analysis, the super-resolution processing, thecontent and the structuring method of the super-resolved texture featuredatabase, and the like are any of the details mentioned in the aboveembodiments and the description thereof is omitted.

It is noted that the present invention is not limited to theconstitution of FIG. 14, and various constitutions are employable. Forexample, the camera 111 may be a part of the client 117.

(Third Constitution Example)

FIG. 15 shows the third constitution example as one example of theconstitution for performing the image processing according to thepresent invention using a mobile camera phone and a TV set. A mobilecamera phone 121 is capable of sending image data to a TV set 124through a network 122 or a memory card 123. The resolution of the mobilecamera phone 121 is lower than the resolution of the TV set 124. Inorder to make the most out of the display ability of the TV set 124, asuper-resolution image is generated through a texture feature quantityanalysis circuit, a super-resolved texture feature database, and animage generation circuit, which are boarded on an internal circuit ofthe TV set, and then, is displayed on the screen. Wherein, the detailsof the texture feature quantity analysis, the super-resolved texturefeature database, and image generation are any of the details mentionedin the above embodiments and the description thereof is omitted.

It is noted that the present invention is not limited to theconstitution of FIG. 15, and various constitutions are employable. Forexample, the mobile camera phone 121 may be a digital still camera or avideo movie camera.

As described above, the present invention is applicable to general videoequipment, such as mobile camera phones, digital still cameras, videomovie cameras, TV sets, in addition to personal computers andserver-client systems, which are all widespread, and requires none ofspecial equipment, operation, management, and the like. Further, thepresent invention imposes no limitation on exclusive hardwareimplementation, combination of software and hardware, and the like,specifically system building method, the aspects of equipmentconnection, the internal construction of the equipment and the like.

The present invention, which generates images having increasedinformation content with no degradation of the image feature, can beutilized in various application fields in which visual informationcontent is placed prime importance on, and various effects are obtained.For example, detailed information on merchandise on which a consumerfocuses attention can be presented in the electronic commerce; precisedetails of exhibits can be presented accurately to viewers in digitalarchives; an ability of video expression in video production isenhanced; and compatibility with various video formats is ensured inbroadcast.

1. An image processing method comprising: a first step of performingimage feature analysis for an original image to obtain an image featurequantity independent from image coordinates; a second step of subjectingthe image feature quantity obtained in the first step to informationdensification to obtain a densified image feature quantity; and a thirdstep of generating, based on the densified image feature quantityobtained in the second step, a densified image in which an informationcontent of the original image is densified.
 2. The image processingmethod of claim 1, wherein the second step includes the steps of:selecting an image feature category to which the image feature quantitybelongs from a plurality of image feature categories prepared inadvance; and reading out a basic image feature quantity, of whichinformation content is densified, in the selected image feature categoryas the densified image feature quantity from a densified image featuredatabase.
 3. The image processing method of claim 1, wherein the secondstep includes the steps of: calculating respective similarities of theimage feature quantity to plural image feature categories prepared inadvance; and summing respective basic image feature quantities in theplural image feature categories with weights according to the respectivecalculated similarities to generate the densified image featurequantity.
 4. The image processing method of claim 1, wherein the secondstep includes the steps of: selecting an image feature category to whichthe image feature quantity belongs from a plurality of image featurecategories prepared in advance; and converting the image featurequantity to the densified image feature quantity by referencing aconversion table database and using a conversion table for featurequantity conversion in the selected image feature category.
 5. The imageprocessing method of claim 4, wherein the plurality of image featurecategories are provided for respective material properties of an objectshot in an image.
 6. The image processing method of claim 1, whereinspatial resolution or time resolution is used as the image featurequantity.
 7. The image processing method of claim 6, wherein spatialfrequency response or time frequency response is obtained using Fouriertransform.
 8. The image processing method of claim 6, wherein spatialfrequency response or time frequency response is obtained using wavelettransform.
 9. The image processing method of claim 6, wherein spatialfrequency response or time frequency response is obtained using aplurality of spatial filters of which at least one of scale, phase, andspatial directionality is different from each other.
 10. An imageprocessing apparatus comprising: an image feature analysis section thatperforms image feature analysis for an original image to obtain an imagefeature quantity independent from image coordinates; an informationdensification section that performs information densification for theimage feature quantity obtained by the image feature analysis section toobtain a densified image feature quantity; and an image generationsection that generates, based on the densified image feature quantityobtained by the information densification section, a densified image inwhich an information content of the original image is densified.
 11. Animage processing program that allows a computer to execute: a first stepof performing image feature analysis for an original image to obtain animage feature quantity independent from image coordinates; a second stepof subjecting the image feature quantity obtained in the first step toinformation densification to obtain a densified image feature quantity;and a third step of generating, based on the densified image featurequantity obtained in the second step, a densified image in which aninformation content of the original image is densified.